Papers with pretraining scheme
Learning to Retrieve Passages without Supervision (2022.naacl-main)
Copied to clipboard
| Challenge: | Dense retrievers for open domain question answering have been shown to achieve impressive performance by training on large datasets of question-passage pairs. |
| Approach: | They propose to use recurring spans to create pseudo examples for contrastive learning. |
| Outcome: | The proposed model outperforms all pretrained baselines on a wide range of ODQA datasets and is competitive with BM25, a strong sparse baseline. |
Few-Shot Question Answering by Pretraining Span Selection (2021.acl-long)
Copied to clipboard
| Challenge: | Pretraining models with recurring span selection are effective, but perform poorly in a few-shot setting. |
| Approach: | They propose recurring span selection scheme that asks model to select correct span in passage with multiple sets of recurring recurrings. |
| Outcome: | The proposed model achieves 72.7 F1 on multiple benchmarks while maintaining competitive performance in the high-resource setting. |
Local Structure Matters Most: Perturbation Study in NLU (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent research shows that neural models are insensitive to word-order perturbations, but other studies suggest that models learn some abstract notion of syntax. |
| Approach: | They develop order-altering perturbations on the order of words, subwords, and characters to analyze their effect on neural models’ performance on language understanding tasks. |
| Outcome: | The proposed models are insensitive to word-order perturbations while the local ordering remains relatively unperturbed. |
Self-Alignment Pretraining for Biomedical Entity Representations (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to self-supervised learning of biomedical entities are limited in the biomedic domain. |
| Approach: | They propose a pretraining scheme that self-aligns the representation space of biomedical entities. |
| Outcome: | The proposed framework achieves state-of-the-art on six MEL benchmarking datasets. |
Segment Any Text: A Universal Approach for Robust, Efficient and Adaptable Sentence Segmentation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Sentence segmentation is a key task in many NLP systems, but no prior method has achieved all of the features needed to segment a text into sentences. |
| Approach: | They propose a new model that uses punctuation to enhance robustness and adaptability. |
| Outcome: | The proposed model outperforms baselines across 8 corpora across diverse domains and languages and is available under the MIT license. |